The Reliability of Measuring Organizational Maturity

One of the recently developed classes of decision making tools for software engineering management is the organizational maturity assessment. The scores from such assessments are being applied in focusing and tracking self-improvement efforts, and as part of the contract award decision making process. However, until now, the important issue of the reliability of assessments has rarely been addressed by the developers and users of assessment methods. The extent of reliability describes the degree to which assessment scores are consistent and repeatable. In this paper we present some basic techniques for the estimation of the reliability of maturity assessments. We then demonstrate through a case study some of these techniques and how to apply the reliability estimate(s) in decision making situations. Examples of decisions are: comparing organizations' maturity scores and evaluating the relationship between maturity scores and some criterion of effectiveness. For the latter, we found a weak relationship betwen maturity and our measure of effectiveness, namely the success of the requirements engineering process.